Skip to main content

Web Crawls

The Web Archive of the Internet Archive started in late 1996, is made available through the Wayback Machine, and some collections are available in bulk to researchers. Many pages are archived by the Internet Archive for other contributors including partners of Archive-IT, and Save Page Now users. Other captures are donated to the Internet Archive by other partners such as Alexa Internet.

5,176,584
RESULTS
rss


Media Type
8,178
collections
5,058,370
web
87,076
movies
21,733
data
479
audio
374
texts
More right-solid
Year
848,333
2018
1,026,819
2017
870,787
2016
559,765
2015
383,492
2014
248,493
2013
More right-solid
Topics & Subjects
1,505,959
wiki
1,457,928
crawldata
1,249,914
dumps
1,190,452
incremental
476,992
Wikipedia
256,198
wikiteam
More right-solid
Collection
More right-solid
Creator
1,223,172
internet archive
1,202,466
wikimedia projects editors
579,307
archive-it
119,699
thumper2.php
73,846
internetarchive
33,026
alexa internet
More right-solid
Language
227,710
Unknown
158,548
English
11,235
German
11,058
Portuguese
9,692
Dutch
9,259
Hungarian
More right-solid
SHOW DETAILS
up-solid down-solid
eye
Title
Date Archived
Creator
Internet Archive Web Crawls
collection
1,180,334
ITEMS
10.5B
VIEWS
collection
eye 10.5B
The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine .
Topic: webwidecrawl
Worldwide Web Crawls
collection
575,083
ITEMS
4.8B
VIEWS
collection
eye 4.8B
Wide crawls of the Internet conducted by Internet Archive. Please visit the Wayback Machine to explore archived web sites. Since September 10th, 2010, the Internet Archive has been running Worldwide Web Crawls of the global web, capturing web elements, pages, sites and parts of sites. Each Worldwide Web Crawl was initiated from one or more lists of URLs that are known as "Seed Lists". Descriptions of the Seed Lists associated with each crawl may be provided as part of the metadata for...
Alexa Crawls
collection
189,259
ITEMS
4.2B
VIEWS
collection
eye 4.2B
Starting in 1996, Alexa Internet has been donating their crawl data to the Internet Archive. Flowing in every day, these data are added to the Wayback Machine after an embargo period.
Topics: web crawl, Alexa
Survey Crawls
collection
88,185
ITEMS
2.4B
VIEWS
collection
eye 2.4B
Survey crawls are run about twice a year, on average, and attempt to capture the content of the front page of every web host ever seen by the Internet Archive since 1996.
Topic: survey crawls
Live Web Proxy Crawls
collection
39,358
ITEMS
1.9B
VIEWS
collection
eye 1.9B
Content crawled via the Wayback Machine Live Proxy mostly by the Save Page Now feature on web.archive.org. Liveweb proxy is a component of Internet Archive’s wayback machine project. The liveweb proxy captures the content of a web page in real time, archives it into a ARC or WARC file and returns the ARC/WARC record back to the wayback machine to process. The recorded ARC/WARC file becomes part of the wayback machine in due course of time.
Archive-It Digital Collection
collection
590,217
ITEMS
1.3B
VIEWS
collection
eye 1.3B
Archive-It is a subscription web archiving service of the Internet Archive that helps organizations harvest, build, and preserve collections of digital content. Partners create domain specific collections of web captures that can be searched on Archive It . Content is hosted and stored at the Internet Archive data centers. Archive-It works with more than 400 partner organizations in 48 U.S. states and 16 countries worldwide including: College and University Libraries State Archives, Libraries,...
Topic: Colleges, Universities, Libraries, Archives, NGOs, Museums
Archive-It Partners
collection
585,756
ITEMS
1.3B
VIEWS
collection
eye 1.3B
Archive-It is the leading web archiving service for collecting and accessing cultural heritage on the web and is a service of Internet Archive used by libraries, archives, governments, non-profits, and other organizations to build collections of web materials.
Topic: TK
Focused Crawls
collection
241,149
ITEMS
973.6M
VIEWS
by Internet Archive
collection
eye 973.6M
Focused crawls are collections of frequently-updated webcrawl data from narrow (as opposed to broad or wide) web crawls, often focused on a single domain or subdomain.
Topic: webcrawl
Fix Broken Links Web Crawls
collection
85,701
ITEMS
749.4M
VIEWS
collection
eye 749.4M
These crawls are part of an effort to archive pages as they are created and archive the pages that they refer to. That way, as the pages that are referenced are changed or taken from the web, a link to the version that was live when the page was written will be preserved. Then the Internet Archive hopes that references to these archived pages will be put in place of a link that would be otherwise be broken, or a companion link to allow people to see what was originally intended by a page's...
collection
eye 748.2M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Custom Crawl Services
collection
83,372
ITEMS
677M
VIEWS
by Internet Archive
collection
eye 677M
Large-scale web harvests and national domain crawls performed for National Libraries, National Archives, preservation partners, research initiatives, and as part of special projects and custom crawling and research services.
Topic: ccs
Top Domains
collection
120,680
ITEMS
669.9M
VIEWS
collection
eye 669.9M
A daily collection of thousands of the most popular web sites according to Alexa.com's top sites rankings .
Topics: daily, popular sites, Alexa
web-group-internal
collection
32,256
ITEMS
585.8M
VIEWS
collection
eye 585.8M
miscellaneous data
Topic: brad tofel
Wide Crawl started April 2013
collection
25,005
ITEMS
563.4M
VIEWS
collection
eye 563.4M
Web wide crawl with initial seedlist and crawler configuration from April 2013.
Archive Team
collection
482,388
ITEMS
549.1M
VIEWS
collection
eye 549.1M
Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history. History is littered with hundreds of conflicts over the future of a community, group, location or...
collection
eye 517.9M
The seed for Wide00014 was: - Slash pages from every domain on the web: -- a list of domains using Survey crawl seeds -- a list of domains using Wide00012 web graph -- a list of domains using Wide00013 web graph - Top ranked pages (up to a max of 100) from every linked-to domain using the Wide00012 inter-domain navigational link graph -- a ranking of all URLs that have more than one incoming inter-domain link (rank was determined by number of incoming links using Wide00012 inter domain links)...
collection
eye 475.3M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Wiki Collections
collection
1,877,298
ITEMS
470.9M
VIEWS
collection
eye 470.9M
Collections of Wiki data
Topics: crawls, data, wiki
Wikipedia Outlinks
collection
32,869
ITEMS
463.6M
VIEWS
collection
eye 463.6M
Crawl of outlinks from wikipedia.org . These files are currently not publicly accessible. from Wikipedia : Wikipedia is a multilingual, web-based, free-content encyclopedia project operated by the Wikimedia Foundation and based on an openly editable model. The name "Wikipedia" is a portmanteau of the words wiki (a technology for creating collaborative websites, from the Hawaiian word wiki, meaning "quick") and encyclopedia. Wikipedia's articles provide links to guide the...
alexa_2007
collection
7,636
ITEMS
451.3M
VIEWS
collection
eye 451.3M
this data is currently not publicly accessible.
Wide Crawl started June 2014
collection
45,313
ITEMS
438.4M
VIEWS
collection
eye 438.4M
Web wide crawl with initial seedlist and crawler configuration from June 2014.
Wayback Indexes
collection
554
ITEMS
422.6M
VIEWS
collection
eye 422.6M
Wayback indexes. This data is currently not publicly accessible.
Wide Crawl Number 12 - started March, 14th 2015
collection
49,621
ITEMS
414.2M
VIEWS
collection
eye 414.2M
Web wide crawl with initial seedlist and crawler configuration from January 2015.
Wikipedia Near Real Time (from IRC)
collection
17,510
ITEMS
363M
VIEWS
collection
eye 363M
This is a collection of web page captures from links added to, or changed on, Wikipedia pages. The idea is to bring a reliability to Wikipedia outlinks so that if the pages referenced by Wikipedia articles are changed, or go away, a reader can permanently find what was originally referred to. This is part of the Internet Archive's attempt to rid the web of broken links .
Topics: Wikipedia, Wikimedia
Wide Crawl started August 2013
collection
21,911
ITEMS
362.3M
VIEWS
collection
eye 362.3M
Web wide crawl with initial seedlist and crawler configuration from August 2013.
Wide Crawl started January 2012
collection
30,362
ITEMS
336.6M
VIEWS
collection
eye 336.6M
Web wide crawl with initial seedlist and crawler configuration from January 2012 using HQ software.
collection
eye 328.2M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
alexa_2006
collection
6,507
ITEMS
323M
VIEWS
collection
eye 323M
this data is currently not publicly accessible.
Wide Crawl started April 2012
collection
39,252
ITEMS
304.3M
VIEWS
collection
eye 304.3M
Web wide crawl with initial seedlist and crawler configuration from April 2012.
collection
eye 284M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Wide Crawl Number 13
collection
46,049
ITEMS
274.9M
VIEWS
collection
eye 274.9M
Web Wide Crawl Number 13
ArchiveBot: The Archive Team Crowdsourced Crawler
collection
16,197
ITEMS
259.2M
VIEWS
collection
eye 259.2M
ArchiveBot is an IRC bot designed to automate the archival of smaller websites (e.g. up to a few hundred thousand URLs). You give it a URL to start at, and it grabs all content under that URL, records it in a WARC, and then uploads that WARC to ArchiveTeam servers for eventual injection into the Internet Archive (or other archive sites). To use ArchiveBot, drop by #archivebot on EFNet. To interact with ArchiveBot, you issue commands by typing it into the channel. Note you will need channel...
Topics: archiveteam, archivebot, webcrawl, robot, love
collection
eye 242.1M
The seed for this crawl was a list of every host in the Wayback Machine This crawl was run at a level 1 (URLs including their embeds, plus the URLs of all outbound links including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Top News
collection
91,559
ITEMS
233.1M
VIEWS
collection
eye 233.1M
A daily collection of hundreds of the world's top news sites.
Topics: daily, news
Wide Crawl started October 2010
collection
15,839
ITEMS
226M
VIEWS
collection
eye 226M
Web wide crawl with initial seedlist and crawler configuration from October 2010
Wide Crawl Started January 2013
collection
15,138
ITEMS
221.8M
VIEWS
collection
eye 221.8M
Wide crawls of the Internet conducted by Internet Archive. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
Wikipedia Outlinks March 2016
collection
27,265
ITEMS
221.5M
VIEWS
collection
eye 221.5M
Crawl of outlinks from wikipedia.org started March, 2016. These files are currently not publicly accessible. Properties of this collection. It has been several years since the last time we did this. For this collection, several things were done: 1. Turned off duplicate detection. This collection will be complete, as there is a good chance we will share the data, and sharing data with pointers to random other collections, is a complex problem. 2. For the first time, did all the different wikis....
Wide Crawl started September 2012
collection
22,402
ITEMS
219.5M
VIEWS
collection
eye 219.5M
Web wide crawl with initial seedlist and crawler configuration from September 2012.
collection
eye 216.1M
Web wide crawl.
Wide Crawl started October 2011
collection
12,648
ITEMS
196.5M
VIEWS
collection
eye 196.5M
Web wide crawl with initial seedlist and crawler configuration from March 2011 using HQ software.
National Library of Australia Crawls
collection
27,720
ITEMS
189.5M
VIEWS
collection
eye 189.5M
Crawls performed by Internet Archive on behalf of the National Library of Australia. This data is currently not publicly accessible.
Wide Crawl started February 2014
collection
9,789
ITEMS
180M
VIEWS
collection
eye 180M
Web wide crawl with initial seedlist and crawler configuration from February 2014.
Wordpress Blogs and the Pages They Link To
collection
22,242
ITEMS
177.8M
VIEWS
collection
eye 177.8M
This is a collection of pages and embedded objects from WordPress blogs and the external pages they link to. Captures of these pages are made on a continuous basis seeded from a feed of new or changed pages hosted by Wordpress.com or by Wordpress pages hosted by sites running a properly configured Jetpack wordpress plugin.
Topics: Wordpress.com, blogs, jetpack
Around The World Crawl
collection
2,150
ITEMS
174.4M
VIEWS
collection
eye 174.4M
Data crawled by Sloan Foundation on behalf of Internet Archive
GDELT
collection
44,553
ITEMS
174M
VIEWS
collection
eye 174M
A daily crawl of more than 200,000 home pages of news sites, including the pages linked from those home pages. Site list provided by The GDELT Project
Topics: GDELT, News
Wide Crawl started March 2011
collection
8,528
ITEMS
172.7M
VIEWS
collection
eye 172.7M
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. What’s in the data set: Crawl start date: 09 March, 2011 Crawl end date: 23 December, 2011 Number of captures: 2,713,676,341 Number of unique URLs: 2,273,840,159 Number of hosts: 29,032,069 The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT)...
.com survey started January 2011
collection
2,535
ITEMS
164M
VIEWS
collection
eye 164M
Survey crawl of .com domains started January 2011.
Topic: webcrawl
alexa_web_2009
collection
3,080
ITEMS
156.4M
VIEWS
collection
eye 156.4M
this data is currently not publicly accessible.
alexa_web_2010
collection
2,994
ITEMS
149.7M
VIEWS
collection
eye 149.7M
this data is currently not publicly accessible.
38_crawl
collection
1,387
ITEMS
136.3M
VIEWS
collection
eye 136.3M
this data is currently not publicly accessible.
Wikipedia Outlinks February 2012
collection
2,951
ITEMS
135.4M
VIEWS
collection
eye 135.4M
Crawl of outlinks from wikipedia.org started February, 2012. These files are currently not publicly accessible.
Archive Team: The News Roundup
collection
31,695
ITEMS
134.9M
VIEWS
by Archive Team
collection
eye 134.9M
Archive Team now searches many, many news sites, including extensive worldwide and obscure sources, to capture unique news stories for history.
web_wk
collection
9,973
ITEMS
126.6M
VIEWS
collection
eye 126.6M
Crawl performed by Internet Archive. This data is currently not publicly accessible.
National Library of Spain Crawls
collection
6,741
ITEMS
117.1M
VIEWS
collection
eye 117.1M
Data collected by Internet Archive on behalf of the National Library of Spain. This data is currently not publicly accessible.
Alexa Crawl EG
collection
1,678
ITEMS
116.2M
VIEWS
collection
eye 116.2M
Crawl EG from Alexa Internet. This data is currently not publicly accessible.
web_iq
collection
2,637
ITEMS
113M
VIEWS
collection
eye 113M
Crawl performed by Internet Archive. This data is currently not publicly accessible.
perma_cc
collection
562,615
ITEMS
110.2M
VIEWS
collection
eye 110.2M
collection
eye 107M
Web wide crawl number 16 The seed list for Wide00016 was made from the join of the top 1 million domains from CISCO and the top 1 million domains from Alexa.
52_crawl
collection
2,589
ITEMS
99.8M
VIEWS
collection
eye 99.8M
this data is currently not publicly accessible.
51_crawl
collection
1,138
ITEMS
94.1M
VIEWS
collection
eye 94.1M
this data is currently not publicly accessible.
26_crawl
collection
1,466
ITEMS
92.7M
VIEWS
collection
eye 92.7M
this data is currently not publicly accessible.
Bibliotheque Nationale de France Domain Crawls
collection
1,652
ITEMS
79.2M
VIEWS
collection
eye 79.2M
Crawls of the french domain space performed by Internet Archive on behalf of Bibliotheque Nationale de France. This data is currently not publicly accessible.
International News Crawls
collection
5,464
ITEMS
75.6M
VIEWS
collection
eye 75.6M
Crawls of International News Sites
35_crawl
collection
1,179
ITEMS
73M
VIEWS
collection
eye 73M
this data is currently not publicly accessible.
Internet Archive Global Events
collection
8,013
ITEMS
69.2M
VIEWS
collection
eye 69.2M
Internet Archive Global Events Archive-It Partner Since: Feb, 2006 Organization Type: Other Institutions Organization URL: http://www.archive-it.org
Alexa Crawl EI
collection
1,408
ITEMS
68.5M
VIEWS
collection
eye 68.5M
Crawl EI from Alexa Internet. This data is currently not publicly accessible.
Survey Crawl Number 6: Sep 11th, 2017 - running now
collection
13,723
ITEMS
68.2M
VIEWS
collection
eye 68.2M
The seeds for this crawl came from: 251 million Domains that had at least one link from a different domain in the Wayback Machine, across all time ~ 300 million Domains that we had in the Wayback, across all time 55,945,067 Domains from https://archive.org/details/wide00016 This crawl was run with a Heritrix setting of "maxHops=0" (URLs including their embeds) The WARC files associated with this crawl are not currently available to the general public.
Shallow Crawls
collection
1,042
ITEMS
67.3M
VIEWS
collection
eye 67.3M
Shallow crawls that collect content 1 level deep including embeds. This data is currently not publicly accessible.
29_crawl
collection
1,568
ITEMS
66.2M
VIEWS
collection
eye 66.2M
this data is currently not publicly accessible.
web_mon
collection
3,809
ITEMS
63.8M
VIEWS
collection
eye 63.8M
Crawl performed by Internet Archive. This data is currently not publicly accessible.
Wikipedia Outlinks May 2011
collection
1,638
ITEMS
63M
VIEWS
collection
eye 63M
Crawl of outlinks from wikipedia.org started May, 2011. These files are currently not publicly accessible.
Alexa Crawl DX
collection
1,442
ITEMS
61.6M
VIEWS
collection
eye 61.6M
Crawl DX from Alexa Internet. This data is currently not publicly accessible.
Elections Web
collection
1,609
ITEMS
60.9M
VIEWS
collection
eye 60.9M
This collection contains collaborative Election crawls performed by IA.
Topics: elections, web
Election Crawl 2012
collection
1,608
ITEMS
60.9M
VIEWS
collection
eye 60.9M
This crawl was performed in Summer & Fall of 2012 to archive the US Federal Elections.
Topics: US, federal, elections, web, 2012
web_el_2008
collection
1,705
ITEMS
58.9M
VIEWS
collection
eye 58.9M
This data is currently not publicly accessible.